Image formats for video capture, processing and display

ABSTRACT

Techniques and tools for representing pixel data in a video processing or capture system are described. Described techniques and tools provide efficient color representation for video processing and capture, and provide flexibility for representing colors using different bit precisions and memory layouts. Described techniques and tools include video formats that can be used, for example, in hardware or software for capture, processing, and display purposes. In one aspect, chroma and luma information for a pixel in a video image is represented in a 16-bit fixed-point block of data having an integer and fractional components. Data can be easily converted from one representation to another (e.g., between 16-bit and 10-bit representations). In other aspects, formats for representing 8-, 10- and 16-bit video image data (e.g., packed and hybrid planar formats), and codes for indicating the formats, are described.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of U.S. patent application Ser. No.10/798,874, filed on Mar. 10, 2004, entitled, “IMAGE FORMATS FOR VIDEOCAPTURE, PROCESSING AND DISPLAY,” now pending, which is hereinincorporated by reference in its entirety.

TECHNICAL FIELD

This invention relates to image formats in video systems, and moreparticularly relates to image formats for pixel data in color spaceshaving luma and chroma channels.

BACKGROUND

In video and imaging systems, color generally is represented as vectorcoordinates in a three dimensional “color space.” Common examplesinclude the well-known classes of RGB and YUV color spaces. The RGBcolor spaces specify pixel values using coordinates that representintensities of red, green and blue light, respectively. The YUV colorspaces specify pixel values using coordinates that represent a luminanceor chrominance value.

Due to the nonlinear response of cathode ray tube (CRT) monitors (alsoknown as the monitor's gamma), data is often specified as intensityvalues relative to the inverse of the gamma. For example, in YUV colorspaces, actual values stored relative to the inverse gamma (typicallydenoted using an apostrophe) are referred to as the “luma” value (Y′)and “chroma” values (U′ and V′). RGB data stored relative to the inversegamma is typically denoted using the notation R′G′B′. This form of datais also known as ‘nonlinear’ YUV or RGB data.

When YUV data is stored in a digital or analog form, the ranges for eachof the components are typically distinguished using a variation of theYUV notation. For example, the notation Y′Pb′Pr′ refers to nonlinear YUVdata where the Y component is in the range of [0 . . . 1] (0 beingblack, 1 being white) and the chroma components range from [−0.5 . . .0.5]. As another example, data stored in an 8-bit or 10-bit fixed pointapproximation is denoted using the notation Y′Cb′Cr′.

For video, pixel data is typically represented using the YUV colorspaces, which can be derived from RGB information. Conversion betweenthe RGB and YUV color spaces typically involves a simple transform. Forexample, the Y′Pb′Pr′ color data is a variation of the R′G′B′ colorspace where the Y′ component is proportional to the perceptualbrightness (Y′=0.299R′+0.587G′+0.114B′), and the Pb′ and Pr′ componentsare defined as color difference from the brightness (e.g., Pb′=B′−Y′;Pr′=R′−Y′).

When Y′Pb′Pr′ is stored in an 8-bit approximation, the Y′ range [0 . . .1] is mapped to the range 16 . . . 235. The chroma components Pb′ andPr′ are mapped to their corresponding Cb′ and Cr′ components by mappingthe range of [−0.5 . . . 0.5] to an interval of width 224 centered about128 (i.e. 16 to 240). For example, an 8-bit representation is computedas:Y′=16+Y′*219Cb′=128+Pb′*224Cr′=128+Pr′*224

Higher bit precisions are computed by simply scaling up the 8-bitvalues. For example, an n-bit representation is computed as:Y′=(16+Y′*219)*2^(n-8)Cb′=(128+Pb′*224)*2^(n-8)Cr′=(128+Pr′*224)*2^(n-8)

Pixel data can be stored in a packed format or planar format. In apacked format, the components corresponding to a given pixel are storedas a cluster or group in a single array in memory, and data for allcomponents can be obtained in a single read. Packed formats can beconvenient when performing operations on an entire pixel, but can beinefficient if an operation on a single channel (e.g., an operation onthe R channel for an RGB pixel) is desired.

For example, in FIG. 1 an RGB pixel in an 8-bit per channelrepresentation is stored in packed format as a little-endian DWORD 100.The data for the RGB pixel comprises 24 bits of color information inadjacent bytes in memory (one byte each for the R, G, and B channels),along with an 8-bit alpha (or transparency) value. Because DWORD 100 islittle-endian, Byte 0 (which contains the alpha value) appears on thefar right as the least significant byte. As shown in FIG. 1, in an 8-bitrepresentation a pixel with three full-resolution channels of colorinformation will have 24 bits of color data. For memory alignment andaccess efficiency reasons, pixel data is often stored on a 16- or 32-bitboundary. Therefore, a pixel with 24 bits of color data is typicallystored along with an 8-bit alpha value, so that the pixel data for eachpixel occupies 32 bits.

In a planar format, information from different channels is stored inseparate planes, rather than being grouped into a single array. Forexample, in FIG. 2 data for an RGB image 200 is stored in an R plane210, a G plane 220, and a B plane 230. In an 8-bit representation, eachpixel would include 8 bits of information from each plane. Planarformats can be inefficient when performing operations on an entire pixelbecause getting data for an entire pixel requires three reads—one foreach plane. However, planar formats can be convenient in othersituations, such as when performing an operation on a single channel(e.g., filtering out all of the red in an RGB image).

Because human perception is not as sensitive to color changes as it isto brightness changes, chroma components can be sampled at a lowerspatial resolution than the luma components with little perceived lossin image quality. For example, in a “4:2:2” sub-sampling format, a pairof chroma samples (e.g., a pair of Cr′ and Cb′ samples) is “shared”between two luma samples. In other words, a 4:2:2 image has one pair ofchroma samples for every two luma samples—the chroma samples aresub-sampled by a factor of two in the horizontal direction.

Table 1 describes several common sub-sampling formats.

TABLE 1 Sub-sampling formats Sub-sampling format Definition 4:4:4 Forevery luma sample, there is a pair of corresponding chroma samples(e.g., a Cb′ and Cr′ sample). 4:2:2 For every two horizontal lumasamples, there is a pair of corresponding chroma samples. (Horizontally,there is half as much chroma information as luma information.) 4:1:1 Forevery four horizontal luma samples, there is a pair of correspondingchroma samples. (Horizontally, there is a quarter as much chromainformation as luma information.) 4:2:0 For each 2 × 2 square of lumasamples, there is a corresponding pair of chroma samples. (Chromainformation is sampled both vertically and horizontally at half theresolution of the luma samples.)

For more information on color spaces and sub-sampling formats, see,e.g., Keith Jack, Video Demystified: A Handbook for the DigitalEngineer, LLH Technology Publishing (3d ed. 2001).

Whatever the benefits of previous techniques, they do not have theadvantages of the following tools and techniques.

SUMMARY

In summary, techniques and tools for representing pixel data in a videoprocessing or capture system are described. Described techniques andtools provide efficient color representation for video processing andcapture, and provide flexibility for representing colors using differentbit precisions and memory layouts. Described techniques and toolsinclude video formats that can be used, for example, in hardware orsoftware for capture, processing, and display purposes.

In one aspect, chroma and luma information for a pixel in a video imageis represented in an n-bit representation comprising a 16-bitfixed-point block of data. The most significant byte in the 16-bit unitof data is an integer component, and the least significant byte in the16-bit unit of data is a fractional component. The n-bit representationis convertible to a lower-precision representation by assigning zerovalues to one or more of the bits in the least significant byte. Forexample, the n-bit representation can be converted to an (n-m)-bitrepresentation by assigning zero values to the m least-significant bitsin the least-significant byte.

In another aspect, chroma and luma information for a pixel in a videoimage is represented in an n-bit representation comprising a 16-bitfixed-point block of data. The most significant byte in the 16-bit unitof data is an integer component, and the least significant byte in the16-bit unit of data is a fractional component. The n-bit representation(e.g., a 10-bit representation) is convertible to a higher-precisionrepresentation (e.g., a 16-bit representation) by changing an identifier(e.g., a FOURCC code) for the video data.

In another aspect, data for a video image is represented in a packedformat representation, the video data consisting of color channel data(e.g., in a YUV color space) and alpha channel data (e.g., 2 bits perpixel, 16 bits per pixel, or some other number of bits per pixel) foreach of plural pixels in the video image. The packed formatrepresentation has a color channel bit precision of greater than eightbits per color channel (e.g., 10 bits per channel, 16 bits per channel,or some other bit precision). The video data can be in differentsub-sampling formats (e.g., 4:4:4, 4:2:2, or some other sub-samplingformat).

In another aspect, pixel data for a video image is represented in apacked format by storing first luma data for a first pixel in a firstunit of memory, storing first chroma data shared by the first pixel anda second pixels in a second unit of memory at a higher memory addressthan the first unit, storing second luma data for the second pixel in athird unit of memory at a higher memory address than the second unit,and storing second chroma data shared by the first and second pixels ina fourth unit of memory at a higher memory address than the third unit.The first and second luma data and the first and second chroma data havea bit precision of greater than 8 bits (e.g., 10 bits, 16 bits or someother number of bits) per channel.

In another aspect, a four-character code is operable to indicate aformat of digital video data in a computer system. The four-charactercode comprises a first character based on whether the format is a packedformat or a hybrid planar format, a second character based on chromasampling (e.g., 4:2:2 sub-sampling, 4:2:0 sub-sampling, etc.) in theformat, and third and fourth characters based on a bit precision (e.g.,8, 10, 16, or some other number of bits per channel). The four charactercode can be included in a file header of a file containing the digitalvideo data. The four-character code can be changed to cast the format ofthe digital video data to a different bit precision.

In another aspect, data for a video image is represented in a hybridplanar format. The hybrid planar format has a bit precision of greaterthan eight bits per channel and includes a first array comprising lumainformation for pixels in the video image and a second array comprisingchroma information for the pixels. The chroma information is storedwithin the second array in a packed format. The hybrid planar format canbe used with different sub-sampling formats.

In another aspect, data for a video image is represented in a hybridplanar format. The hybrid planar format has a bit precision of greaterthan or equal to eight bits per channel. The hybrid planar formatincludes a first array comprising luma information for pixels in thevideo image and a second array comprising chroma information for thepixels. The chroma information is stored within the second array in apacked format and is sub-sampled in the horizontal direction by a factorof two.

The various techniques and tools can be used in combination orindependently. Different embodiments implement one or more of thedescribed techniques and tools.

Additional features and advantages will be made apparent from thefollowing detailed description of different embodiments that proceedswith reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an RGB pixel in an 8-bit per channelrepresentation stored in packed format as a little-endian DWORD.

FIG. 2 is a block diagram showing a planar format for an RGB image.

FIG. 3 is a block diagram of a suitable computing environment forimplementing techniques and tools for video image formats.

FIG. 4 is a block diagram of a computer system for reading andprocessing video data.

FIG. 5 is a block diagram showing a little-endian WORD representation ofa 16-bit channel.

FIG. 6 is a block diagram showing a little-endian WORD representation ofa 10-bit channel.

FIG. 7 is a flow diagram of a technique for converting a 16-bitfixed-point representation of pixel data into a 10-bit representation.

FIG. 8 is a block diagram showing a memory layout for the P016 and P010formats.

FIG. 9 is a block diagram showing a memory layout for the P216 and P210formats.

FIG. 10 is a block diagram showing a memory layout for the P208 format.

FIG. 11 is a block diagram showing a bit layout of a pair of Y216 orY210 pixels.

FIG. 12 is a block diagram showing a bit layout of a little-endian DWORDencoding a Y410 pixel.

FIG. 13 is a block diagram showing a bit layout of a little-endian QWORDencoding a Y416 pixel.

FIG. 14 is a block diagram showing an AYUV memory layout.

FIG. 15 is a block diagram showing an A144 memory layout.

FIG. 16 is a block diagram showing a YUY2 memory layout.

FIG. 17 is a block diagram showing UYVY a memory layout.

FIG. 18 is a block diagram showing an NV12 memory layout.

FIG. 19 is a block diagram showing an NV11 memory layout.

DETAILED DESCRIPTION

The following description is directed to techniques and tools forrepresenting pixel data in a video processing or capture system.Described techniques and tools provide efficient color representationfor video processing and capture, and provide flexibility forrepresenting colors using different bit precisions (e.g., 8, 10 or 16bits per channel).

Described techniques and tools include image representation formats thatcan be used, for example, in hardware or software for capture,processing, and display purposes. For example, different packed formatsand planar formats (e.g., hybrid planar formats) are described.

For example, in a described embodiment, color is represented in a 10-bithybrid planar format in a YUV color space. The 10-bit hybrid planarformat involves a plane of luma (Y) information combined with packedchroma (U and V) pairs. The 10-bit hybrid planar format allows efficientvideo processing because of its hardware-friendly memory layout.

Described techniques and tools provide an efficient and simple mechanismto switch between different bit precisions and to define alpha channels.Described techniques and tools can be applied to several differentsub-sampling formats (e.g., 4:4:4, 4:2:2, 4:2:0 and 4:1:1). Differentformats offer different advantages in terms of visual quality, bandwidthand/or processing efficiency.

I. Computing Environment

The techniques and tools described above can be implemented on any of avariety of computing devices and environments, including computers ofvarious form factors (personal, workstation, server, handheld, laptop,tablet, or other mobile), distributed computing networks, and Webservices, as a few general examples. The techniques and tools can beimplemented in hardware circuitry, as well as in software executingwithin a computer or other computing environment, such as shown in FIG.3.

FIG. 3 illustrates a generalized example of a suitable computingenvironment 300 in which described techniques and tools can beimplemented. The computing environment 300 is not intended to suggestany limitation as to scope of use or functionality of the invention, asthe present invention may be implemented in diverse general-purpose orspecial-purpose computing environments.

With reference to FIG. 3, the computing environment 300 includes atleast one processing unit 310 and memory 320. In FIG. 3, this most basicconfiguration 330 is included within a dashed line. The processing unit310 executes computer-executable instructions and may be a real or avirtual processor. In a multi-processing system, multiple processingunits execute computer-executable instructions to increase processingpower. The memory 320 may be volatile memory (e.g., registers, cache,RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), orsome combination of the two. The memory 320 stores software 380implementing described techniques and tools for video image formats.

A computing environment may have additional features. For example, thecomputing environment 300 includes storage 340, one or more inputdevices 350, one or more output devices 360, and one or morecommunication connections 370. An interconnection mechanism (not shown)such as a bus, controller, or network interconnects the components ofthe computing environment 300. Typically, operating system software (notshown) provides an operating environment for other software executing inthe computing environment 300, and coordinates activities of thecomponents of the computing environment 300.

The storage 340 may be removable or non-removable, and includes magneticdisks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or any othermedium which can be used to store information and which can be accessedwithin the computing environment 300. For example, the storage 340stores instructions for implementing software 380.

The input device(s) 350 may be a touch input device such as a keyboard,mouse, pen, or trackball, a voice input device, a scanning device, oranother device that provides input to the computing environment 300. Forvideo, the input device(s) 350 may be a video capture device (e.g., adigital video camera) or other device that accepts input in analog ordigital form. The output device(s) 360 may be a display, printer,speaker, CD-writer, or another device that provides output from thecomputing environment 300.

The communication connection(s) 370 enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,audio/video or other media information, or other data in a modulateddata signal. By way of example, and not limitation, communication mediainclude wired or wireless techniques implemented with an electrical,optical, RF, infrared, acoustic, or other carrier.

Techniques and tools described herein can be described in the generalcontext of computer-readable media. Computer-readable media are anyavailable media that can be accessed within a computing environment. Byway of example, and not limitation, with the computing environment 300,computer-readable media include memory 320, storage 340, andcombinations of any of the above.

Some techniques and tools herein can be described in the general contextof computer-executable instructions, such as those included in programmodules, being executed in a computing environment on a target real orvirtual processor. Generally, program modules include functions,programs, libraries, objects, classes, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. The functionality of the program modules may be combined or splitbetween program modules as desired. Computer-executable instructions maybe executed within a local or distributed computing environment.

II. Image Formats for Video Capture, Processing, and Display

Described techniques and tools are directed to different surface formatsfor video data. Described techniques and tools are useful fordisplaying, capturing, and processing video in a computer system (e.g.,a PC). Described techniques and tools provide efficient hardwareprocessing and easy conversion between data of different bit precisions(e.g., between 10- and 16-bit representations). Described surfaceformats are associated with new FOURCC codes (i.e. 32-bit identifiertags) which indicate the format of the video data. Although many of thedescriptions herein refer only to YUV formats for simplicity, describedtechniques and tools also can be applied to the Y′Cb′Cr color formats,or to other color spaces. In addition, although specific implementationsinvolving 8-, 10- and 16-bit representations are described, describedtechniques and tools can be modified to include other higher or lowerbit precisions.

In the descriptions herein, the term “surface” refers to an area inmemory (e.g., video memory or system memory) where pixel data is stored.The “surface origin” is the upper left corner (i.e., pixel coordinates(0, 0)) of the surface. The “stride” of the surface (sometimes calledthe “pitch”) is the width of the surface in bytes. Although the strideis a signed integer, given a surface origin at the upper-left corner,the stride is always positive. The “alignment” of a surface refers tothe byte position at which lines in a surface originate. Typically, thealignment of the surface is at the discretion of the graphics displaydriver and its underlying hardware. For example, in some hardwareimplementations, a surface must always be DWORD aligned, that is,individual lines within a surface must originate on a DWORD (i.e.,four-byte or 32-bit) boundary. Other alignments are possible, dependingon implementation. For example, an alignment can be greater than 32bits, depending on the needs of the hardware. In general, a WORD is adata unit consisting of two bytes, a DWORD is a data unit consisting offour bytes, and a QWORD is a data unit consisting of eight bytes.

If present, an alpha value indicates transparency. For example, in someimplementations an alpha value of 0 indicates a completely transparentimage, whereas an alpha value of (2^n)−1 indicates an opaque image,where n is the number of alpha bits. Alpha is assumed to be a linearvalue that is applied to each component after it has been converted intoits normalized linear form.

A. FOURCC Codes

FOURCC codes are used to identify different video data formats. A FOURCCcode is a 32-bit unsigned integer created by concatenating four ASCIIcharacters. Typically, a FOURCC code is stored in a file (e.g., in afile header) along with video data. The FOURCC code identifies theformat of the video data. The video hardware and/or software thatprocesses the information can determine whether the video data is in acompatible format based on the FOURCC code. For example, in FIG. 4 acomputer system 400 for reading and processing video data takes a videodata file 410 as input into a file reader 420. The file reader 420 sendsvideo data including a FOURCC code to a video renderer 430. Typically,the video renderer 430 determines whether the video data is compatiblewith the system based on the FOURCC code. If it is compatible, the videodata is sent to the GPU 440 for processing and subsequent display oncomputer display 450.

Described techniques and tools include surface formats havingcorresponding FOURCC codes derived according to certain conventions. Innaming conventions described herein, the first character of the FOURCCcode indicates the packed and/or planar layout of the video data. Thesecond character indicates a sub-sampling format (e.g., 4:4:4, 4:2:2,4:2:0, 4:1:1). The third and forth characters indicate a bit precision(e.g., 8-bit, 10-bit, or 16-bit representation).

For example, in one embodiment, hybrid planar surface format codes startwith the uppercase character literal “P.” “P” is used to represent thefact that all the surface formats are planar in layout for the Ycomponent. The packed surface format codes start with the uppercaseliteral “Y.” The next character in the FOURCC code is determined by thechroma sub-sampling scheme, as shown in Table 2 below:

TABLE 2 FOURCC code characters based on sub-sampling scheme Sub-samplingscheme FourCC code letter 4:4:4 ‘4’ 4:2:2 ‘2’ 4:1:1 ‘1’ 4:2:0 ‘0’The final two character literals are used to indicate whether thesurface contains 8-, 10- or 16-bit data. 16-bit data is represented bythe character literals “16,” 10-bit data is represented by “10,” and8-bit data is represented by “08.” Alternatively, one or more of thesespecific characters can be replaced with other characters in accordancewith the naming conventions without affecting the ability of the FOURCCcodes to effectively indicate the formats to which they correspond.

Several formats described in detail below have FOURCC codes that complywith the naming conventions described above. P016 and P010 are FOURCCcodes for 16- and 10-bit versions of a hybrid planar 4:2:0 format; P216,P210 and P208 are FOURCC codes for 16-, 10- and 8-bit versions of ahybrid planar 4:2:2 format; Y216 and Y210 are FOURCC codes for 16- and10-bit versions of a packed 4:2:2 format; and Y416 and Y410 are FOURCCcodes for 16- and 10-bit versions of a packed 4:4:4 format. Other codesand formats also can be used. For example, although formatscorresponding to the codes P116 and P110 (which would correspond to 16-and 10-bit versions of a hybrid planar 4:1:1 format) are not describedin detail, such codes formats can be used in alternativeimplementations.

B. 16-Bit and 10-Bit Fixed-Point Representations

Described embodiments include techniques and tools for implementing 10-and 16-bit formats (e.g., in the YUV color space) using a fixed-pointrepresentation for the both the luma channel and the chroma channels.The 16-bit formats use little-endian WORDs of data. For example, FIG. 5shows a little-endian WORD representation 500 of a 16-bit channel. InFIG. 5, an integer component 510 consists of bits 8-15, and a fractionalcomponent 520 consists of bits 0-7.

Described 10-bit formats also assign 16 bits to each channel. However,in the 10-bit formats, the lower order six bits of each little-endianWORD are cleared to zero. This allows data in a 16-bit format to beprocessed by existing 10-bit hardware, because the 10-bit system cansimply drop the last 6 bits of the WORD. For example, FIG. 6 shows alittle-endian WORD representation 600 of a 10-bit channel. In FIG. 6,the integer component 610 consists of bits 8-15, but only bits 6 and 7are used in the fractional component 620. The lower six bits of thefractional component 620 are cleared to zero.

The memory layout of the 10-bit and 16-bit representations make itpossible to cast a 10-bit representation of YUV data to a 16-bitrepresentation (e.g., by adding 0s to the lower six bits of thefractional component) without loss of precision. The memory layout alsomakes it possible to cast a 16-bit representation to a 10-bitrepresentation by dropping the lower six bits of the fractionalcomponent. When hardware reads a surface that contains a 10-bitrepresentation, it can ignore the lower-order 6 bits. If the surfacecontains valid 16-bit data, it should be identified as a 16 bit surface.FOURCC codes can be used to identify 16-bit and 10-bit representationsand to indicate how to appropriately process the video data and/or castthe video data into different representations.

FIG. 7 shows an exemplary technique 700 for converting between differentbit precisions. In particular, FIG. 7 shows a technique converting a16-bit fixed-point representation of pixel data into a 10-bitrepresentation. At 710, pixel data in 16-bit fixed point representationis received. Then, at 720, the data is converted to 10-bitrepresentation by dropping the lower six bits of the fractionalcomponent. Based on the described memory layouts, other techniques forconverting between different bit precisions also can be used (e.g.,converting from 10- to 16-bit precisions, converting from 16- to 8-bitprecisions, etc.)

SMPTE standard 274-1998 defines semantics on how to handle 10-bit and16-10 bit forms of YUV as extensions of 8-bit data. In someimplementations, the new “n” bit formats store the sample values as8-bit values scaled by the factor 2^(n-8) in accordance with SMPTE274-1998 sections 7.7, 7.8, 7.11 and 7.12. Precision conversions can beperformed using simple shifts. For example, if a white point for an8-bit format was at value 235, then the corresponding 10-bitrepresentation would have a white point at 235*4=940.

Described 10- and 16-bit formats are advantageous because the data ismost-significant-bit justified (i.e. the upper 8 bits always correspondto the ‘integer’ portion of the original fraction). The SMPTE standardassumes that higher bit representations of the same practical value areleft shifts (i.e. multiples of 2) of the base value.

C. New Format Definitions

Techniques and tools for implementing new formats for video image dataare described. The described formats for storing representations ofvideo image data are defined in such a way that they are efficientlyprocessed by video capture hardware, CPUs, video processing hardware(e.g., graphics processing units (GPUs)) and video display hardware(e.g., digital-to-analog converters (DACs)), and can be easily supportedon current and future devices. In some cases, existing RGB hardwaresupport can be re-used to implement described techniques and tools.

1. Hybrid Planar Format Definitions

Hybrid planar formats are formats having characteristics of both packedformats and planar formats. For example, in formats described hereinluma information is stored separately from chroma information, with lumasamples stored in one plane and chroma samples stored in packed formatwithin another plane. Image operations can be performed separately onthe luma and chroma planes. For example, the data for a YUV image isstored as a plane of Y samples followed by a plane of packed U and Vsamples. The pair of Y and U/V planes are often concatenated vertically,and the format can be referred to as a “hybrid planar” YUV surfaceformat.

a. 4:2:0 Hybrid Planar Formats: P016 and P010

P016 and P010 are FOURCC codes for 4:2:0 16- and 10-bit hybrid planarrepresentations, respectively. All Y samples are found first in memoryas an array of little-endian WORDs with an even number of lines(possibly with a larger stride for memory alignment), followedimmediately by an array of little-endian WORDs containing interleaved Uand V samples.

The combined U-V array is addressed as an array of little-endian DWORDvalues (i.e. a U-V pair is a DWORD). The least significant WORD (“LSW”)of each pair contains the U value and the most significant WORD (“MSW”)contains the V value. The combined U-V array has the same total strideas the array of Y samples, but there are half as many lines ofinterleaved U-V samples as there are Y samples because the chromachannels are sub-sampled by a factor of two in the vertical dimension.

For example, FIG. 8 shows a memory layout 800 for the P016 and P010formats. FIG. 8 shows units of memory labeled Y0, Y1, Y2, Y3 in lumaarray 810 and U0, V0, U2 and V2 in chroma array 820. These labeled unitsof memory each represent a single little-endian WORD. In P016 and P010,the luma and chroma samples are in 16-bit representation and 10-bitrepresentation, respectively. The U and V samples are packed: in eachU-V DWORD pair, a one-WORD U sample is followed by a one-WORD V sample.

In one implementation, the P016 and P010 formats are preferred 4:2:0planar pixel formats for 16- and 10-bit precision representations insystems supporting 16- and/or 10-bit 4:2:0 video.

b. 4:2:2 Hybrid Planar Formats: P216, P210 and P208

P216 and P210 are FOURCC codes for 4:2:2 16- and 10-bit representations,respectively, where all Y samples are found first in memory as an arrayof little-endian WORDs with an even number of lines (possibly with alarger stride for memory alignment), followed immediately by an array oflittle-endian WORDs containing interleaved U and V samples.

The combined U-V array is addressed as an array of little-endian DWORDvalues (i.e. a U-V pair is a DWORD). The LSW of each pair contains the Uvalue and the MSW contains the V value. The combined U-V array has thesame total stride as the array of Y samples, and there are as many linesof interleaved U-V samples as there are lines of Y samples. For example,FIG. 9 shows a memory layout 900 for the P216 and P210 formats. FIG. 9shows units of memory labeled Y0, Y1, Y2, Y3 in luma array 910 and U0,V0, U1 and V2 in chroma array 920.

In one implementation, the P216 and P210 formats are preferred 4:2:2hybrid planar pixel formats for 16- and 10-bit precision representationsin systems supporting 16- and/or 10-bit 4:2:2 video.

P208 is a FOURCC code for a 4:2:2 8-bit representation where all Ysamples are found first in memory as an array of bytes. The array'swidth is a multiple of two (possibly with a larger stride for memoryalignment) and is followed immediately by an array of byte pairscontaining interleaved U and V samples. The stride of the array of U-Vpairs must also be divisible by two.

The combined U-V array is addressed as an array of little-endian WORDvalues (i.e. a U-V pair is a WORD) with the same total stride as thearray of Y samples. The least-significant byte of each pair (“LSB”)contains the U value and the most-significant byte (“MSB”) contains theV value. Horizontally, there are exactly half as many interleaved U-Vsamples as there are Y samples. For example, FIG. 10 shows a P208 memorylayout 1000. FIG. 10 shows units of memory labeled Y0, Y1, Y2, Y3, Y4,Y5, Y6 and Y7 in luma array 1010 and U01, V01, U23, V23, U45, V45, U67and V67 in chroma array 1020. Each of these units of memory is a byte.

P208 is advantageous over packed 4:2:2 formats, especially foroperations such as video encoding or decoding which require searching oraccessing information on two independent planes (the luma plane and thechroma plane). (For further advantages of described hybrid planarformats, see section I.C.1.c. below)

In one implementation, the P208 format is a preferred 4:2:2 hybridplanar pixel format for 8-bit 4:2:2 representations.

c. Advantages of Described Hybrid Planar Formats

In some implementations, hybrid planar formats can achieve 5%-17% orbetter improvements in CPU processing (as compared with arrangementswhere chroma components are split into two separate planes). The factorsinvolved in the improved processing include:

Smaller cache usage and better locality. With split planes (e.g., twochroma planes), each pixel fetch requires three cache locations—one forthe luma plane, and one each for each chroma component plane. But whenthe chroma components are paired within the same plane, one chroma cacheline holds both components, and only requires one fetch. Thiseffectively cuts cache usage in half, since performing fetches on twoplanes uses twice as much cache. The cache line holds half as manypixels, but tends to contain spatially closer values that can be morereadily used. When searching for neighboring pixels, similar behaviorapplies—more data around the pixel of interest tends to be read whenpulling in the cache line of component pairs rather than two separatecache lines, and the cost of fetching unused “peripheral” pixels isreduced.

Tiled memory coherency. Video cards tend to tile their memory in 2×2blocks (128 bit cache line tiles) and tile 256×256 regions into 8×8“macroblock” tiles. Paired chroma components align well with 2×2 tiles.

Padding semantics. For images with a stride, padding semantics arewell-defined. For example, in 4:2:0 and 4:2:2 formats, the chromaplane's data is the same width as the luma plane because the chroma datais sub-sampled by a factor of two, but the components are paired.Therefore, stride computations are simple and allow for arbitrary linealignments. On the other hand, split chroma panes put padding in themiddle and right edge of an image, and do not allow for “uncommitted”memory (i.e. memory which is not physically present or accessible on theleft side of the image. Split panes therefore have less support forunlockable or selectively lockable regions of the image. When locking animage, padding on the right hand side of an image may point touncommitted memory, or memory in a tiled region which has a heavy accesscost. If the padding is split between the middle of the image and theright side, the memory locking model does not work unless the size ofthe middle padding is made to be the same as the unlockable region onthe right, which is a wasteful use of memory.

For example, if a line must be a multiple of 256 bytes, for an image of720×480 the stride would be 768 bytes (thereby “wasting” 48 bytes perscanline). However, in 4:2:2 and 4:2:0 formats, after rounding up theluma plane lines to 768 bytes the chroma plane lines would be 384 byteswide. To make the chroma plane lines a multiple of 256 bytes wouldrequire them to be rounded up from 384 bytes to 512 bytes. This would inturn require the luma plane to be rounded up to 1024 bytes. Thus, forthe luma and chroma plane lines each be a multiple of 256 bytes, anadditional 256 bytes are ‘wasted’ for every scanline.

Stride alignment. Image stride alignments are half as coarse, resultingin significant memory savings for standard definition content. Videohardware typically allocates memory at 256-byte boundaries due to 8×8“macroblock” tiling. With modern CPUs (e.g., Intel CPUs with SSE2technology), cache usage can be better controlled. Doubling pixel widthincreases cache alignments, and power-of-2 alignments fit better withcache lines.

Parallelization. When chroma data is of a lower spatial samplingfrequency than luma data, using two planes of data (e.g., a luma planefollowed by a plane of chroma pairs) naturally partitions the image intotwo independently accessible images, with one image (luma) at fullresolution and another image (chroma) down-sampled from the fullresolution (e.g., ½ or ¼ of the full resolution). This provides theopportunity for optimizations or simplifications for software andhardware processing algorithms to operate on the data with twoindependent operations (or even operations executable in parallel).Split chroma planes would require two overlapping chroma operations tobe performed simultaneously, which may cause cache coherencydegradations or interdependencies with multiple processing entities.

Amortized computational costs on chroma components. With split chromaplanes, filtering coefficients, loops, etc. are typically identical whenapplied to each chroma plane. By working on the chroma components inpairs, the same computations (e.g., application of filteringcoefficients) can be applied to both chroma components and only computedonce for each pair of components.

2. Packed Format Definitions

In a packed format, the color data for one or more pixels are stored inconsecutive bytes in the same array. Depending on implementation,processing pixels stored in a packed format may involve operating onindividual pixels, pairs of pixels, or some other set of pixels. Forexample, in a 4:2:2 packed format in a YUV color space, a pair of chromasamples is shared by two pixels. Luma samples from the two differentpixels are interleaved with the pair of chroma samples, and reading thedata of one pixel would therefore require reading the data of bothpixels. Data reads must therefore be performed on “pairs” of pixels inthe 4:2:2 packed format.

a. 4:2:2 Packed Formats: Y210 and Y216

Y216 and Y210 are the FOURCC codes for a packed 16-bit and 10-bit 4:2:2format, respectively. Each individual Y, U and V value is encoded as asingle little-endian WORD. Pairs of pixels are stored together in anarray of four little endian WORDs. Y210 is identical to Y216 except thatthere are only 10 significant bits of data for each channel value, whichare most-significant-bit justified in a 16-bit little endian WORD (i.e.only bits 15-6 are used; bits 5-0 are 0). For example, FIG. 11 shows abit layout 1100 of a pair of Y216 or Y210 pixels. The WORDs labeled Y0,U, Y1 and V are packed together in a single array.

b. 4:4:4 Packed Formats: Y416 and Y410

Y410 is the FOURCC code for a packed 10-bit 4:4:4 representation thatincludes a two-bit alpha value. In Y410, each pixel is encoded as asingle little endian DWORD with the individual Y, U, V and alphachannels laid out as shown in FIG. 12. The bit layout 1200 in FIG. 12shows a little-endian DWORD encoding a Y410 pixel. The four bytes in theDWORD are labeled Byte 0, Byte 1, Byte 2 and Byte 3. The U channel takesup Byte 0 and two bits of Byte 1, the Y channel takes up six bits ofByte 1 and four bits of Byte 2, the V channel takes up four bits of Byte2 and six bits of Byte 3, and the alpha channel (A) takes up theremaining two bits of Byte 3. Y410 allows the three 10-bit luma andchroma channels to be stored in one DWORD, with a two-bit alpha channelincluded for a 32-bit alignment. In one implementation, applications setthe two alpha channel bits to the value 0x03 to indicate that the pixelis fully opaque.

Y416 is the FOURCC code for a packed 16-bit 4:4:4 representation thatincludes a 16-bit alpha value. In one implementation, the Y416 format isan intermediate format intended to avoid error accumulation inprocessing. Each pixel is encoded as a single little-endian QWORD withthe individual Y, U, V and alpha channels laid out as shown in FIG. 13.This format is similar to a 16-bits per channel version of Y410. The bitlayout 1300 in FIG. 13 shows a little-endian QWORD encoding a Y416pixel. Because of the way a little-endian QWORD is used in the physicalmemory, the memory layout in FIG. 13 is “byte swapped.” The U channeltakes up WORD 0, the Y channel takes up WORD 1, the V channel takes upWORD 2, and the alpha channel (A) takes up WORD 3. The 16-bit alphachannel is included for a 64-bit alignment. In one implementation,applications set the alpha channel bits to the value 0xffff to indicatethe pixel is fully opaque.

D. Existing Format Definitions

Existing formats can be used in combination with one or more of thedescribed new formats or other techniques and tools. Brief descriptionsof some of these formats and their corresponding FOURCC codes areprovided below.

AYUV is a 4:4:4 packed format where each pixel is encoded as fourconsecutive bytes. FIG. 14 shows an AYUV memory layout 1400. Eachlabeled memory unit in FIG. 14 (V0, U0, etc.) is a byte.

AI44 is a 4:4:4, 16-entry paletted format with four bits of alpha perpixel. Each pixel is stored in a byte as four bits of index in the uppernibble and four bits of alpha in the lower nibble. FIG. 15 shows an AI44memory layout 1500. Each labeled memory unit in FIG. 15 (Index 0, Alpha0, etc.) is a nibble.

YUY2 and UYVY are 4:2:2 packed formats where each “macropixel” is twopixels encoded as four consecutive bytes, with chroma informationdown-sampled in the horizontal direction by a factor of two. FIG. 16shows a YUY2 memory layout 1600. In FIG. 16, the first byte of the“macropixel” contains the first Y sample (Y0), the second byte containsthe first U sample (U01), the third byte contains the second Y sample(Y1), and the fourth byte contains the first V sample (V01). The UYVYformat is the same as YUY2 except that the byte pairs are exchanged.FIG. 17 shows a UYVY memory layout 1700.

NV12 is an 8-bit 4:2:0 format where all Y samples are found first inmemory as an array of bytes with an even number of lines (possibly witha larger stride for memory alignment), followed immediately by an arrayof byte pairs containing interleaved U and V samples. The combined U-Varray is addressed as an array of little-endian WORD values (i.e. a U-Vpair is a WORD). The least significant byte of each pair (“LSB”)contains the U value and the most significant byte (“MSB”) contains theV value with the same total stride as the Y samples. There are exactlyhalf as many lines of interleaved U-V samples as there are Y samples.FIG. 18 shows an NV12 memory layout 1800. Each labeled memory unit inFIG. 18 (U0, V0, etc.) is a byte. In one implementation, NV12 is apreferred format for 8-bit 4:2:0 representations.

NV11 is an 8-bit 4:1:1 representation where all Y samples are foundfirst in memory as an array of bytes whose width is a multiple of four(possibly with a larger stride for memory alignment), followedimmediately by an array of byte pairs containing interleaved U and Vsamples. The stride must also be divisible by two. The combined U-Varray is addressed as an array of little-endian WORD values (i.e. a U-Vpair is a WORD). The LSB contains the U value and the MSB contains the Vvalue with the same total stride as the Y samples. Horizontally, thereare exactly one quarter as many interleaved U-V samples as there are Ysamples. FIG. 19 shows an NV11 memory layout 1900. Each labeled memoryunit in FIG. 19 (U0-3, V0-3, etc.) is a byte. In one implementation,NV11 is a preferred format for 8-bit 4:1:1 representations.

E. Example Preferred Formats in One Implementation

In one implementation, different formats are preferred depending oncharacteristics of video data to be processed, desired visual quality ofvideo, and/or capabilities of a computerized video system. In thisexemplary implementation, the following formats (indicated by FOURCCcodes) are preferred formats (preferred formats indicated as being“planar” have luma information in a separate plane from chromainformation):

-   -   AI44—16-entry indexed paletted format with 4 bits alpha    -   AYUV—8-bit packed 4:4:4 format    -   YUY2 (or UYVY)—8-bit packed 4:2:2 format    -   NV11—8-bit planar 4:1:1 format    -   NV12—8-bit planar 4:2:0 format    -   P208—8-bit planar 4:2:2 format    -   Y210, Y216—packed 4:2:2 format (10-bit & 16-bit)    -   Y410, Y416—packed 4:4:4 format (10-bit & 16-bit)    -   P210, P216—planar 4:2:2 format (10-bit & 16-bit)    -   P010, P016—planar 4:2:0 format (10-bit & 16-bit)

Other implementations can support additional formats, omit some formats,or replace one or more of the listed formats with other formats.

Having described and illustrated the principles of our invention withreference to the described techniques and tools, it will be recognizedthat the described techniques and tools can be modified in arrangementand detail without departing from such principles. It should beunderstood that the programs, processes, or methods described herein arenot related or limited to any particular type of computing environment,unless indicated otherwise. Various types of general purpose orspecialized computing environments may be used with or performoperations in accordance with the teachings described herein. Elementsof the described implementations shown in software may be implemented inhardware and vice versa.

In view of the many possible embodiments to which the principles of ourinvention may be applied, we claim as our invention all such embodimentsas may come within the scope and spirit of the following claims andequivalents thereto.

1. A method of displaying a video image by a video processing systemhaving a processor and a display, the method comprising: the processorprocessing video data to produce the video image, the video data havinga packed format representation, the video data consisting of colorchannel data and alpha channel data for each of plural pixels in thevideo image, the packed format representation having the color channeldata and alpha channel data of each pixel arranged in a single bit arrayper group of at least one pixel, the packed format representationfurther having a color channel bit precision of greater than eight bitsper color channel, and the packed format representation beingrepresented by a total number of bits of the single bit array which isaligned with an integer multiple of an 8-bit byte but is not equal to aninteger multiple of the color channel bit precision; and displaying thevideo image on the display.
 2. The method of claim 1 wherein the colorchannel data has a bit precision of 10 bits per channel.
 3. The methodof claim 1 wherein the alpha channel data consists of 2 bits per pixel.4. The method of claim 1 where the video data is in a 4:2:2 sub-samplingformat.
 5. The method of claim 1 where the video data is in a 4:4:4sub-sampling format.
 6. The method of claim 1 wherein the color channeldata is in a YUV color space.
 7. A computer-readable storage mediumhaving instructions stored thereon executable by a computer systemhaving a processor and a display for performing a method of displaying avideo image, the method comprising: the processor processing video datato produce the video image, the video data having a packed formatrepresentation, the video data consisting of color channel data andalpha channel data for each of plural pixels in the video image, thepacked format representation having the color channel data and alphachannel data arranged in a single bit array per group of at least onepixel, the packed format representation further having a color channelbit precision of greater than eight bits per color channel, and thepacked format representation being represented by a total number of bitsof the single bit array which is aligned with an integer multiple of an8-bit byte but is not equal to an integer multiple of the color channelbit precision; and displaying the video image on the display.
 8. Thecomputer-readable storage medium of claim 7 wherein the color channeldata has a bit precision of 10 bits per channel.
 9. Thecomputer-readable storage medium of claim 7 wherein the alpha channeldata consists of 2 bits per pixel.
 10. The computer-readable storagemedium of claim 7 where the video data is in a 4:2:2 sub-samplingformat.
 11. The computer-readable storage medium of claim 7 where thevideo data is in a 4:4:4 sub-sampling format.
 12. The computer-readablestorage medium of claim 7 wherein the color channel data is in a YUVcolor space.